dat_full <- preprocessing(root_dir, data_dir)
dat <- subset(dat_full, subset=RT>0 & RT<25000) # loses 2 RTs that were judged to be outliers 25978 49871
dat_borderline <- dat
dat <- perSubjectTrim.fnc(dat, response='RT', subject='Subject', trim = 2.5)$data
## ......
## n.removed = 204 
## percent.removed = 2.797203
dat$RT_log <- log(dat$RT)

Info and Procedure (Method)

The main aim of this experiment was to see whether vague instructions would confer advantages over crisp alternatives when all instructions used numerals, and when there were vague and crisp versions of each of the comparison and matching strategies.

On each trial, first the referring expression that constituted the instruction for that trial was displayed (e.g., “Choose a square with about 10 dots”). Participants then pressed a key to indicate that they had read the instruction.

The instruction remained on screen, and after 1000 ms, the arrays appeared (see Figure below).

Response time was measured from the presentation of the arrays until the keypress indicating the participant’s choice. The trial would timeout after 60 seconds if there was no response.

In this experiment, no feedback was given. This was because, in the vague conditions, we did not regard any response as “correct” or “incorrect”, but instead as “expected response”; “borderline response”; and “extreme response”, and we did not want to draw participants’ attention to this distinction explicitly. Which choice the participant made was recorded for analysis.

Full table of instructions

We changed “fewer than” (crisp) to “far fewer than” (in vague): this was because “far fewer than” can have borderline cases whereas “fewer than” cannot have borderline cases.

Full table of instructions
Item Quantity Selection Crisp Vague
06:15:24 Small Comparison Choose a square with fewer than 20 dots Choose a square with far fewer than 20 dots
06:15:24 Small Matching Choose a square with 6 dots Choose a square with about 10 dots
06:15:24 Large Comparison Choose a square with more than 10 dots Choose a square with far more than 10 dots
06:15:24 Large Matching Choose a square with 24 dots Choose a square with about 20 dots
16:25:34 Small Comparison Choose a square with fewer than 30 dots Choose a square with far fewer than 30 dots
16:25:34 Small Matching Choose a square with 16 dots Choose a square with about 20 dots
16:25:34 Large Comparison Choose a square with more than 20 dots Choose a square with far more than 20 dots
16:25:34 Large Matching Choose a square with 34 dots Choose a square with about 30 dots
26:35:44 Small Comparison Choose a square with fewer than 40 dots Choose a square with far fewer than 40 dots
26:35:44 Small Matching Choose a square with 26 dots Choose a square with about 30 dots
26:35:44 Large Comparison Choose a square with more than 30 dots Choose a square with far more than 30 dots
26:35:44 Large Matching Choose a square with 44 dots Choose a square with about 40 dots
36:45:54 Small Comparison Choose a square with fewer than 50 dots Choose a square with far fewer than 50 dots
36:45:54 Small Matching Choose a square with 36 dots Choose a square with about 40 dots
36:45:54 Large Comparison Choose a square with more than 40 dots Choose a square with far more than 40 dots
36:45:54 Large Matching Choose a square with 54 dots Choose a square with about 50 dots

Means plots

dat_plot <- summarySEwithin2(dat, measurevar="RT_log", withinvars=c("Vagueness", "Selection", "Item"), idvar="Subject")

Hypotheses

We formulated the following hypotheses:

Results

Response times were trimmed at 25,000 ms, resulting in the loss of 2 trials that were judged to have outlier RTs. The remaining data were trimmed separately for each participant at 2.5 standard deviations below and above that participant’s mean RT, leading to the loss of 204 trials (2.8% of the remaining trials). RTs in milliseconds were log-transformed so that they better approximated a normal distribution for the purposes of analysis.

The first set of models

dat_model <- dat
dat_model$c_Vag <- ifelse(dat_model$Vagueness=="Crisp", -0.5, 0.5)
dat_model$c_Sel <- ifelse(dat_model$Selection=="Comparison", -0.5, 0.5)
dat_model$c_Itm <- ifelse(dat_model$Item=="06:15:24", -.75, ifelse(dat_model$Item=="16:25:34", -.25, ifelse(dat_model$Item=="26:35:44", .25, .75)))
rtlmer = lmerTest::lmer(data=dat_model, RT_log ~ c_Vag * c_Sel + c_Itm + (1 + c_Vag * c_Sel + c_Itm | Subject))
pretty_coef_table(rtlmer, "rt_lmer_full_model_all_items")
rt_lmer_full_model_all_items
term 𝛽   s.e.  d.f. 𝑡  𝑝   Pr(>|t|)  sig.
c_Vag -0.0051 0.014 36.94 -0.37 7.16e-01 0.716
c_Sel 0.1619 0.026 36.99 6.24 2.97e-07 <0.001 ＊＊＊
c_Itm 0.0580 0.018 36.92 3.23 2.60e-03 <0.01  ＊＊
c_Vag:c_Sel 0.1324 0.031 36.87 4.23 1.49e-04 <0.001 ＊＊＊ 
dat_model <- subset(dat, subset=Selection=='Comparison')
dat_model$c_Vag <- ifelse(dat_model$Vagueness=="Crisp", -0.5, 0.5)
dat_model$c_Itm <- ifelse(dat_model$Item=="06:15:24", -.75, ifelse(dat_model$Item=="16:25:34", -.25, ifelse(dat_model$Item=="26:35:44", .25, .75)))
comp_lmer <- lmerTest::lmer(data=dat_model, RT_log ~ c_Vag + c_Itm + (1 + c_Vag + c_Itm | Subject))
pretty_coef_table(comp_lmer, "rt_lmer_for_comparison_all_items")
rt_lmer_for_comparison_all_items
term 𝛽   s.e.  d.f. 𝑡  𝑝   Pr(>|t|)  sig.
c_Vag -0.071 0.020 36.99 -3.52 1.15e-03 <0.01  ＊＊
c_Itm 0.090 0.020 37.02 4.61 4.68e-05 <0.001 ＊＊＊ 
dat_model <- subset(dat, subset=Selection=='Matching')
dat_model$c_Vag <- ifelse(dat_model$Vagueness=="Crisp", -0.5, 0.5)
dat_model$c_Itm <- ifelse(dat_model$Item=="06:15:24", -.75, ifelse(dat_model$Item=="16:25:34", -.25, ifelse(dat_model$Item=="26:35:44", .25, .75)))
match_lmer <- lmerTest::lmer(data=dat_model, RT_log ~ c_Vag + c_Itm + (1 + c_Vag + c_Itm | Subject))
pretty_coef_table(match_lmer, "rt_lmer_for_matching_all_items")
rt_lmer_for_matching_all_items
term 𝛽   s.e.  d.f. 𝑡  𝑝   Pr(>|t|)  sig.
c_Vag 0.062 0.021 36.32 2.91 6.14e-03 <0.01  ＊＊
c_Itm 0.025 0.022 36.62 1.12 2.71e-01 0.271

Considering the first set of models:

  • [Hypothesis 1] Crisp/Vague RT:
    • Vague instructions should result in faster responses than crisp instructions : Vague instructions resulted in faster responses than crisp instructions but the difference was not significant (p=0.716).
  • [Hypothesis 2] Comparison/Matching RT:
    • Instructions that allow comparison should result in faster responses than instructions that necessitate matching : comparison instructions resulted in faster responses than matching instructions, and the difference was significant (p<0.001).
  • [Hypothesis 3] Vagueness effect differs between comparison and matching
    • The effects of vagueness should differ when the selection task is comparison versus when it is matching : Vagueness exerted effects in different directions for the two selection tasks (p<0.001). Separate analyses at each level of selection provided evidence that vague instructions resulted in faster responses than crisp instructions for the comparison condition (p<0.01); and slower responses than crisp instructions in the matching condition (p<0.01).

The second set of models

The second set of models considered were conducted to test for main effects in the presence of interactions involving those main effects (after Levy, 2018).

The variables are: Vagueness 2 levels (X) Item 4 levels (Y) Selection 2 levels (Z)

Note that the null-hypothsesis model includes random slopes for the main effect of X; Y; and Z; as well as for their interactions – including a seperate random slope for all combinations of X, Y, and Z achieves this.

dat_model <- dat
dat_model$c_Vag <- ifelse(dat_model$Vagueness=="Crisp", -0.5, 0.5)
dat_model$c_Sel <- ifelse(dat_model$Selection=="Comparison", -0.5, 0.5)
dat_model$c_Itm <- ifelse(dat_model$Item=="06:15:24", -.75, ifelse(dat_model$Item=="16:25:34", -.25, ifelse(dat_model$Item=="26:35:44", .25, .75)))

Item.numeric <- sapply(dat_model$Item,function(i) contr.sum(4)[i,])
dat_model$Item1 <- Item.numeric[1,]
dat_model$Item2 <- Item.numeric[2,]
dat_model$Item3 <- Item.numeric[3,]
null.model2 <- lmer(RT_log ~ 
                     c_Sel + c_Vag:c_Sel + # main effect of selection and the interaction with vagueness
                     Item1 + c_Vag:Item1 +         # item effect and interaction with vagueness
                     Item2 + c_Vag:Item2 +         # item effect and interaction with vagueness
                     Item3 + c_Vag:Item3 +         # item effect and interaction with vagueness
                     (c_Sel + c_Vag + c_Itm| Subject), # per-subject
                   dat_model, REML=FALSE) 
pretty_coef_table(null.model2, "null.model2")
null.model2
term 𝛽   s.e.  d.f. 𝑡  𝑝   Pr(>|t|)  sig.
c_Sel 0.1595 0.0249 38.00 6.41 1.56e-07 <0.001 ＊＊＊
Item1 -0.0677 0.0137 54.80 -4.93 8.09e-06 <0.001 ＊＊＊
Item2 0.0096 0.0095 937.85 1.01 3.15e-01 0.315
Item3 0.0329 0.0095 939.36 3.45 5.83e-04 <0.001 ＊＊＊
c_Sel:c_Vag 0.1307 0.0205 6943.95 6.38 1.89e-10 <0.001 ＊＊＊
c_Vag:Item1 0.1020 0.0178 6941.32 5.74 9.71e-09 <0.001 ＊＊＊
c_Vag:Item2 -0.0311 0.0177 6939.89 -1.75 8.02e-02 0.08
c_Vag:Item3 -0.0555 0.0178 6939.84 -3.13 1.78e-03 <0.01  ＊＊ 
full.model2 <- lmer(RT_log ~ 
                     c_Vag + 
                     c_Sel + c_Vag:c_Sel + # main effect of selection and the interaction with vagueness
                     Item1 + c_Vag:Item1 +         # item effect and interaction with vagueness
                     Item2 + c_Vag:Item2 +         # item effect and interaction with vagueness
                     Item3 + c_Vag:Item3 +         # item effect and interaction with vagueness
                     (c_Sel + c_Vag + c_Itm| Subject), # per-subject
                   dat_model, REML=FALSE) 
pretty_coef_table(full.model2, "full.model2")
full.model2
term 𝛽   s.e.  d.f. 𝑡  𝑝   Pr(>|t|)  sig.
c_Vag -0.0057 0.0137 37.93 -0.42 6.78e-01 0.678
c_Sel 0.1618 0.0255 37.99 6.34 1.93e-07 <0.001 ＊＊＊
Item1 -0.0660 0.0143 52.93 -4.60 2.65e-05 <0.001 ＊＊＊
Item2 0.0102 0.0096 808.66 1.05 2.92e-01 0.292
Item3 0.0324 0.0096 809.21 3.36 8.17e-04 <0.001 ＊＊＊
c_Vag:c_Sel 0.1306 0.0205 6942.69 6.38 1.95e-10 <0.001 ＊＊＊
c_Vag:Item1 0.1020 0.0178 6941.29 5.74 9.74e-09 <0.001 ＊＊＊
c_Vag:Item2 -0.0311 0.0177 6939.87 -1.75 8.01e-02 0.08
c_Vag:Item3 -0.0555 0.0178 6939.82 -3.13 1.78e-03 <0.01  ＊＊ 
anova(null.model2, full.model2)
## Data: dat_model
## Models:
## null.model2: RT_log ~ c_Sel + c_Vag:c_Sel + Item1 + c_Vag:Item1 + Item2 + 
## null.model2:     c_Vag:Item2 + Item3 + c_Vag:Item3 + (c_Sel + c_Vag + c_Itm | 
## null.model2:     Subject)
## full.model2: RT_log ~ c_Vag + c_Sel + c_Vag:c_Sel + Item1 + c_Vag:Item1 + 
## full.model2:     Item2 + c_Vag:Item2 + Item3 + c_Vag:Item3 + (c_Sel + c_Vag + 
## full.model2:     c_Itm | Subject)
##             Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
## null.model2 20 8559.6 8696.9 -4259.8   8519.6                         
## full.model2 21 8561.4 8705.6 -4259.7   8519.4 0.1742      1     0.6764
  • [Hypothesis 1] Crisp/Vague RT:
    • Vague instructions should result in faster responses than crisp instructions : Vague instructions resulted in faster responses than crisp instructions on average (beta = -0.0057 units of log(ms)). However this difference was not significant in the full model (p=0.678). Using Levy’s method to test for main effects in the presence of higher-order interactions, by doing model comparison between a null model that included all interaction terms involving Vagueness but leaving out a term for the main effect of Vagueness, against a full model that differed only by including Vagueness as a main effect, showed that the full model was no better than the reduced model (df=1, p=0.6764) - consituting more evidence that Vagueness did not exert a significant main effect on response times.
  • [Hypothesis 2] Comparison/Matching RT:
    • Instructions that allow comparison should result in faster responses than instructions that necessitate matching : comparison instructions resulted in faster responses than matching instructions, and the difference was significant (beta=0.1618, p<0.001).
  • [Hypothesis 3] Vagueness effect differs between comparison and matching
    • The effects of vagueness should differ when the selection task is comparison versus when it is matching : Although Vagueness did not exert a significant main effect, Vagueness did exert effects in interactions with some other variables: the interaction between Vagueness and Selection task was significant (beta=0.1206, p<0.001), suggesting that Vagueness speeded RTs in the comparison condition but slowed them down in the matching task. Separate analyses at each level of selection provided evidence that vague instructions resulted in faster responses than crisp instructions for the comparison condition (p<0.01); and slower responses than crisp instructions in the matching condition (p<0.01).